Migration to Pydantic v2: Enable compatibility with later FastAPI versions by ChrisChapman-gh · Pull Request #5017 · microsoft/AzureTRE

ChrisChapman-gh · 2026-07-31T08:26:11Z

This PR migrates the Azure TRE codebase from Pydantic v1.10.19 to v2.13.4 to enable compatibility with later versions of FastAPI that require Pydantic v2.

Overview

Later versions of FastAPI require Pydantic v2, and this migration ensures Azure TRE can upgrade FastAPI without being blocked by Pydantic version constraints.

Key Changes

🔧 Core Infrastructure Updates

Requirements updated: Both api_app/requirements.txt and airlock_processor/requirements.txt now specify Pydantic v2.13.4
Backward compatibility: backwards compatibility for data written/read from cosmos or from templates in v1 must be maintained.

🏗️ Model Architecture Migration

Base model modernized: AzureTREModel now uses Pydantic v2 ConfigDict with v1 fallback
Configuration migration: allow_population_by_field_name → populate_by_name
Validator updates: Migrated from @validator to @field_validator with compatibility layer

📦 Component Updates

API App: 21 files updated including domain models and schemas
Airlock Processor: Added compatibility layer for parse_obj_as → TypeAdapter pattern
Schema modernization: Applied automated updates using bump-pydantic tool

Example Migration Pattern

Before (Pydantic v1):

from pydantic import BaseConfig, BaseModel, validator

class AzureTREModel(BaseModel):
    class Config(BaseConfig):
        allow_population_by_field_name = True
        arbitrary_types_allowed = True

    @validator("etag", pre=True)
    def parse_etag(cls, value):
        return value.replace('"', '')

After (Pydantic v2 with v1 compatibility):

try:
    # Pydantic v2
    from pydantic import BaseModel, ConfigDict, field_validator
    
    class AzureTREModel(BaseModel):
        model_config = ConfigDict(
            populate_by_name=True,
            arbitrary_types_allowed=True
        )
        
    @field_validator("etag", mode="before")
    @classmethod
    def parse_etag(cls, value):
        return value.replace('"', '')
        
except ImportError:
    # Pydantic v1 fallback
    from pydantic import BaseConfig, BaseModel, validator
    # ... v1 implementation

Testing & Validation

✅ Comprehensive test suite: All existing functionality preserved
✅ FastAPI compatibility: Confirmed working with FastAPI 0.115.3
✅ Component isolation: API app and airlock processor independently validated
✅ Migration tools: Used official bump-pydantic tool for schema updates

Impact

Files changed: 23 files total (410 additions, 442 deletions)
Net code reduction: Cleaner, more modern Pydantic v2 patterns
Zero breaking changes: Maintains all existing API contracts
Future-ready: Enables FastAPI upgrades requiring Pydantic v2

Migration Benefits

Unblocks FastAPI upgrades - Later FastAPI versions require Pydantic v2
Performance improvements - Pydantic v2 offers significant performance gains
Better type safety - Enhanced validation and serialization capabilities
Modern patterns - Cleaner configuration and validation syntax

Fixes #4637.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

…ility layer Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

…grate .dict() to .model_dump() Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

…n, and .dict() calls Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

- Remove all try/except blocks providing Pydantic v1 fallback support - Update imports to use only Pydantic v2 (TypeAdapter instead of parse_obj_as) - Clean up TypeAdapter usage throughout codebase - Fix syntax errors and whitespace issues - Maintain all existing functionality with Pydantic v2 patterns Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

….8.6->0.9.0 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

…paces.py Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

… compatability with v1

- add explicit defaults for nullable model fields - restore User-to-dict conversion for persisted resource actors - migrate remaining serialization to model_dump() - fix nested Event Grid payload serialization - restore removed resource history behavior and route imports - preserve legacy role ID, optional email, and cost date handling - update tests for Pydantic v2 response types and error messages

…tedResource classes

Good catch from copilot, we can assume that v1 and v2 will not be installed at the same time and that for imports - this is superfluous Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…/resources.py:50) now removes any nested $id with a non-empty URI fragment, including #/properties/..., #properties/..., and absolute URI fragments. Root $id, valid nested IDs, and the original schema object remain unchanged. [test_resource_repository.py (line 413)](/workspaces/AzureTRE/api_app/tests_ma/test_db/test_repositories/test_resource_repository.py:413) covers all three invalid forms, including the exact firewall value.

Copilot

Pull request overview

Copilot reviewed 118 out of 118 changed files in this pull request and generated no new comments.

Suppressed comments (3)

api_app/models/schemas/airlock_request_url.py:16

The OpenAPI example uses the key "container_url", but the response model field is "containerUrl" (camelCase). This makes the generated docs misleading for clients.
api_app/models/schemas/workspace_service.py:29
The OpenAPI example uses "workspace_service", but the response model field is "workspaceService". This mismatch can confuse API consumers reading the docs.
api_app/models/schemas/user_resource.py:34
The OpenAPI example uses "user_resource", but the response model field is "userResource". Align the example key with the actual response shape.

github-actions · 2026-08-03T13:11:47Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/30644447349 (with refid f090555c)

(in response to this comment from @ChrisChapman-gh)

JC-wk

I can't see the v1 compatibility code that is described in the PR ? @ChrisChapman-gh @marrobi
commit ebc7766 ("Remove Pydantic v1 backward compatibility") removed all try/except fallback blocks.

JC-wk

LGTM
I would suggest a future update to display a warning that a templates has failed v2 validation saying v1/legacy template support will be removed in version x and that they should upgrade their templates. And then tighten up and areas that were done for v1 compatibility reasons.

marrobi · 2026-08-03T14:17:00Z

                        polling_count = 0

                    async with credentials.get_credential_async_context() as credential:
-                        service_bus_client = ServiceBusClient(config.SERVICE_BUS_FULLY_QUALIFIED_NAMESPACE, credential)


@JC-wk @ChrisChapman-gh should this be in this PR? Is it not in a different PR?

marrobi

Some issues flagged by an AI review.

marrobi · 2026-08-03T14:22:08Z

-    authorizedRoles: Optional[List[str]] = Field(default=[], title="If not empty, the user is required to have one of these roles to install the template")
-    properties: Dict[str, Property] = Field(title="Template properties")
+    authorizedRoles: Optional[List[str]] = Field(default_factory=list, title="If not empty, the user is required to have one of these roles to install the template")
+    properties: Dict[str, Any] = Field(title="Template properties")


ResourceTemplate.properties went from Dict[str, Property] to Dict[str, Any]. This means registered template properties are now completely unvalidated. Was this deliberate, or a workaround for a validation failure? Given it lands alongside remove_legacy_null_property_fields, it feels like the latter. If it's intentional it needs a comment explaining why; if it's a workaround, I'd like to know what was actually breaking.

fixed - see commit: 61a28d4

marrobi · 2026-08-03T14:23:33Z

-    value: Union[dict, str] = Field(None, title="value", description="value to use in substitution for the property to update")
-    arraySubstitutionAction: Optional[str] = Field("", title="Array Substitution Action", description="How to treat existing values of this property in an array [overwrite | append | replace | remove]")
-    arrayMatchField: Optional[str] = Field("", title="Array match field", description="Name of the field to use for finding an item in an array - to replace/remove it")
+    value: Union[dict, str] = Field(default=None, title="value", description="value to use in substitution for the property to update")


The default doesn't satisfy the annotation. v2 won't complain (defaults aren't validated unless validate_default=True), but this should be Optional[Union[dict, str]] = Field(default=None, ...) — otherwise the declared type is a lie for any step that omits value.

fixed in 9059b0a5ff7f240d9161c2bcd1c2c832c8ea05dd

marrobi · 2026-08-03T14:24:18Z

@@ -92,9 +92,16 @@ class Operation(AzureTREModel):
    message: str = Field("", title="Additional operation status information")
    createdWhen: float = Field("", title="POSIX Timestamp for when the operation was submitted")


"" is not a float. Passes silently in v2, but any Operation constructed without these gets a str where downstream code expects a number. Since almost every other field in this file was touched, worth fixing to 0.0 in the same pass. Line 49 (resourceTemplateName: Optional[str] = Field("")) and line 90 (status: Status = Field(None)) have the same smell.

fixed in 9059b0a5ff7f240d9161c2bcd1c2c832c8ea05dd

marrobi

More feedback:

In AirlockRequestInCreate, remove the empty-string default from type so callers must provide a valid AirlockRequestType (import or export).

In AirlockReviewInCreate, remove the empty-string default from approval so callers must provide a boolean. Update the OpenAPI example from the string "True" to the boolean true.

Preserve intentional const: null and default: null values. Update the legacy schema cleanup so it does not remove const or default when their value is null; both are valid JSON Schema and can carry intentional semantics. Add tests proving that const: null continues to reject non-null values and that default: null remains present after template enrichment. Retain coverage for removing only the legacy null fields that are genuinely invalid or generated accidentally.

Correct invalid typed defaults across all models touched by this migration. Audit every changed Pydantic model for defaults that do not satisfy the annotated type, including empty strings assigned to enums, booleans, floats, or integers, and None assigned to non-optional fields. For required fields, remove the default; for genuinely optional fields, use Optional[...] with None; otherwise use a valid value of the declared type. Pay particular attention to AirlockRequestInCreate.type, AirlockReviewInCreate.approval, PipelineStepProperty.value, Operation.status, Operation.createdWhen, Operation.updatedWhen, and similar fields. Add tests that instantiate models with omitted fields and verify they either fail validation when required or produce values matching their declared types.

properties was changed from Dict[str, Property] to Dict[str, Any] to fix a jsonschema.SchemaError caused by the legacy Property model serialising optional fields as null (e.g. "items": null), which is invalid in JSON Schema. Three changes make Dict[str, Property] safe again: - Property.model_config adds extra="allow" so unknown JSON Schema keywords ($ref, oneOf, format, if/then/else, etc.) are preserved rather than silently dropped on deserialisation. - Property type field is made Optional[str] so properties that use $ref or const without an explicit type are accepted. - Property gains a @model_serializer(mode='plain') that emits only explicitly- set fields (model_fields_set), excludes None values, and recurses into nested plain-dict sub-schemas (items, properties) to strip any legacy null values. A hasattr guard handles the edge case where Pydantic calls the serialiser with an uncoerced plain dict due to item-level dict assignment bypassing validate_assignment. ResourceTemplate gains validate_assignment=True so direct field assignment coerces dict values to Property instances, and a @model_serializer(mode='wrap') that calls _strip_none_recursive on the full serialised output to cover allOf and other plain-dict fields that Pydantic's exclude_none does not recurse into. The legacy remove_legacy_null_property_fields function and its LEGACY_NULL_PROPERTY_FIELDS allowlist in schema_service are removed; null sanitisation is now owned by the model layer. The test was reproducing that step to put the mock enriched_template_mock.return_value into the same state it would be in after enrich_template had run. Now that ResourceTemplate._serialize calls _strip_none_recursive, allOf: None is stripped during model_dump() itself — so neither the guard in enrich_template nor the pop in the test is needed. The pop("allOf", None) is now a no-op and can be removed entirely

Copilot

Pull request overview

Copilot reviewed 118 out of 118 changed files in this pull request and generated no new comments.

Suppressed comments (2)

api_app/models/domain/resource_template.py:13

This recursive cleanup removes every None value, including valid JSON Schema annotations and constraints such as "default": null and "const": null. Checked-in schemas already use null defaults (for example templates/workspace_services/ohdsi/template_schema.json:59), and a user-supplied const: null would be silently deleted, broadening validation. Restrict cleanup to keywords for which null is structurally invalid rather than deleting arbitrary null-valued schema entries.
airlock_processor/StatusChangedQueueTrigger/init.py:13
Under Pydantic v2, Optional[str] without a default is still required. RequestProperties.previous_status was optional under v1 but now rejects status events that omit it, even though the API's corresponding StatusChangedData.previous_status defaults to None. Preserve the existing queue contract by giving this field a None default.

from pydantic import BaseModel, TypeAdapter

ChrisChapman-gh · 2026-08-04T08:21:28Z

More feedback:

In AirlockRequestInCreate, remove the empty-string default from type so callers must provide a valid AirlockRequestType (import or export).

In AirlockReviewInCreate, remove the empty-string default from approval so callers must provide a boolean. Update the OpenAPI example from the string "True" to the boolean true.

Preserve intentional const: null and default: null values. Update the legacy schema cleanup so it does not remove const or default when their value is null; both are valid JSON Schema and can carry intentional semantics. Add tests proving that const: null continues to reject non-null values and that default: null remains present after template enrichment. Retain coverage for removing only the legacy null fields that are genuinely invalid or generated accidentally.

Correct invalid typed defaults across all models touched by this migration. Audit every changed Pydantic model for defaults that do not satisfy the annotated type, including empty strings assigned to enums, booleans, floats, or integers, and None assigned to non-optional fields. For required fields, remove the default; for genuinely optional fields, use Optional[...] with None; otherwise use a valid value of the declared type. Pay particular attention to AirlockRequestInCreate.type, AirlockReviewInCreate.approval, PipelineStepProperty.value, Operation.status, Operation.createdWhen, Operation.updatedWhen, and similar fields. Add tests that instantiate models with omitted fields and verify they either fail validation when required or produce values matching their declared types.

Fixed the first two in 9059b0a5ff7f240d9161c2bcd1c2c832c8ea05dd

… and set default status

Copilot

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated no new comments.

Suppressed comments (3)

api_app/models/domain/resource_template.py:42

Pydantic v2 no longer coerces numeric or boolean values to strings by default, so keeping enum as List[str] makes registration reject valid JSON Schemas such as {"enum": [1, 2]}. JSON Schema enum values may be heterogeneous; preserve them as Any values instead.
airlock_processor/StatusChangedQueueTrigger/init.py:13
Under Pydantic v2, Optional[str] without a default is still a required field. The previous v1 parser accepted status-change messages that omitted previous_status, but this new TypeAdapter path rejects them before processing; give the field a None default to preserve that input contract.

from pydantic import BaseModel, TypeAdapter

api_app/requirements.txt:25

The PR description promises a Pydantic v1/v2 compatibility fallback, but this hard pin and the direct use of v2-only APIs (ConfigDict, field_validator, and TypeAdapter) make the application v2-only. Either implement the documented fallback or update the migration/rollback expectations to describe a hard cutover.

…pdate enum type in Property model

Copilot

Pull request overview

Copilot reviewed 122 out of 122 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

api_app/requirements.txt:25

The PR description promises a Pydantic v1/v2 compatibility layer, but this hard pin and the unconditional use of v2-only APIs (ConfigDict, field_validator, TypeAdapter, and model_dump) make the application unable to import under Pydantic v1. Either implement and test the documented fallback across both components or update the PR description to state that this is a v2-only migration.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 122 out of 122 changed files in this pull request and generated no new comments.

Suppressed comments (2)

api_app/models/domain/resource.py:113

value still has a default of None, so Output(name=..., type=...) is accepted. This directly contradicts the new test_output_requires_value test and allows malformed deployment outputs through validation. Make the field required while retaining Optional only if an explicit JSON null is valid.
api_app/requirements.txt:25
The PR description explicitly promises a Pydantic v1 fallback, but this hard pin (along with unconditional use of v2-only APIs such as TypeAdapter, field_validator, and model_dump) makes both components v2-only. Either implement the documented compatibility path or update the PR description and migration claims so operators do not expect a rolling v1/v2 transition.

ChrisChapman-gh requested a review from a team as a code owner July 31, 2026 08:26

Copilot AI balanced review requested due to automatic review settings July 31, 2026 08:26

Copilot AI reviewed Jul 31, 2026

Copilot AI and others added 27 commits July 31, 2026 08:27

Initial plan

921193b

Implement Pydantic v1/v2 compatibility layer and update requirements

3d9a5ef

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Complete Pydantic v2 migration for airlock processor and add compatib…

594a1a4

…ility layer Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Fix Pydantic v2 compatibility issues: update field annotations and mi…

e53fdbc

…grate .dict() to .model_dump() Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Complete Pydantic v2 migration: fix parse_obj_as usage and test fixtures

7691059

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Fix remaining Pydantic v2 issues: TypeAdapter imports, user validatio…

2d02ba1

…n, and .dict() calls Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Increment component versions: API 0.24.5->0.25.0, Airlock Processor 0…

35fbe58

….8.6->0.9.0 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Fix linting issues: remove unused imports from template route files

05b23fb

Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Fix unit test errors: remove double .model_dump() calls in test_works…

d09fead

…paces.py Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

WIP

7741805

Remove some of the compatibility code.

acb0bd8

Switch to User objects

0e3b5d9

Simplify serialization.

4274043

Add tests back

de29df7

update tests

2c15ddd

Updates to simplify.

93978e8

Updates to simplify code.

0e82758

Fix user model

da52f9f

Update models to correct syntax for v2

057e6c7

Address comments.

17b8f05

Update pydantic version and refactor user fields to maintain backward…

d8462cd

… compatability with v1

Remove leftover merge marker from changelog

376959b

Bump version to 0.8.12 for airlock_processor and 0.25.28 for api_app

19b99a9

Remove unnecessary newline at the end of ResourceTemplate and Restric…

c359c48

…tedResource classes

Apply suggestions from code review

626e15a

Good catch from copilot, we can assume that v1 and v2 will not be installed at the same time and that for imports - this is superfluous Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings August 3, 2026 12:56

Copilot started reviewing on behalf of ChrisChapman-gh August 3, 2026 12:56 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

JC-wk reviewed Aug 3, 2026

View reviewed changes

JC-wk approved these changes Aug 3, 2026

View reviewed changes

marrobi reviewed Aug 3, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 3, 2026 20:47

Copilot started reviewing on behalf of ChrisChapman-gh August 3, 2026 20:48 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Fix, PR comments and feedback

7cf47a2

ChrisChapman-gh added 3 commits August 4, 2026 08:22

fix: update AirlockRequest and Operation models to use Optional types…

9059b0a

… and set default status

fix defaults

33d6a17

update schema examples to present correct types.

98dc11c

Copilot AI review requested due to automatic review settings August 4, 2026 09:30

Copilot started reviewing on behalf of ChrisChapman-gh August 4, 2026 09:31 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

fix: set default value for previous_status in RequestProperties and u…

30e9c83

…pdate enum type in Property model

Copilot AI review requested due to automatic review settings August 4, 2026 10:14

Copilot started reviewing on behalf of ChrisChapman-gh August 4, 2026 10:15 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Comment thread api_app/models/domain/resource.py Outdated

Potential fix for pull request finding

3a0e0e4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings August 4, 2026 10:43

Copilot started reviewing on behalf of ChrisChapman-gh August 4, 2026 10:43 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

		@@ -92,9 +92,16 @@ class Operation(AzureTREModel):
		message: str = Field("", title="Additional operation status information")
		createdWhen: float = Field("", title="POSIX Timestamp for when the operation was submitted")

Uh oh!

Conversation

ChrisChapman-gh commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Changes

🔧 Core Infrastructure Updates

🏗️ Model Architecture Migration

📦 Component Updates

Example Migration Pattern

Testing & Validation

Impact

Migration Benefits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

JC-wk left a comment

Choose a reason for hiding this comment

Uh oh!

JC-wk left a comment

Choose a reason for hiding this comment

Uh oh!

marrobi Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

marrobi left a comment

Choose a reason for hiding this comment

Uh oh!

marrobi Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisChapman-gh Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

marrobi Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisChapman-gh Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marrobi Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisChapman-gh Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marrobi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ChrisChapman-gh commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

ChrisChapman-gh commented Jul 31, 2026 •

edited

Loading

ChrisChapman-gh Aug 4, 2026 •

edited

Loading

ChrisChapman-gh Aug 4, 2026 •

edited

Loading

ChrisChapman-gh commented Aug 4, 2026 •

edited

Loading